Boot-window metrics — record turns/cache-creation instead of gating on magic thresholds, surface per-PR delta#470
Merged
Conversation
added 4 commits
July 2, 2026 20:33
…nance Removes the two absolute 60k threshold branches from assertShallowBootMeasuredTurns (calibrated once against a since-superseded baseline) and records the greet turn's full token usage as a distinct shallow-boot-window journeymetrics observation instead, so boot cost rides a trend instead of a magic-number gate. Adds RunID/RunURL/CapturedAt provenance fields to journeymetrics.Record, stamped at EmitRecord time, and fixes release.yml's journey-ledger job to download each discovered Runtime Live E2E run into its own subdirectory (instead of flat-copying, which silently collapsed multiple runs' same-named records into one observation) and to widen discovery to every successful run since the previous release tag of any kind.
Adds `spacedock-release journey-delta`, which selects the latest-by-captured_at observation per scenario/model from the previously published release ledger, diffs it against the current PR run's freshly emitted records, and posts the result as a single sticky-updating PR comment (gh pr comment --edit-last). Wires a new journey-delta-comment job into runtime-live-e2e.yml, gated to pull_request events with its own pull-requests: write permission.
Adds two spacedock-release CLI primitives the AC-4 backfill runbook composes: shallow-boot-window-record extracts a shallow-boot-window observation from an archived claude-stream.jsonl (reusing AC-1's extraction logic, hoisted out of its test file into internal/ensigncycle so a non-test binary can import it), and ledger-diff proves a rebuilt journey-costs ledger equals the backed-up original plus exactly the named added scenarios before any `gh release upload --clobber` runs. Both subcommands get CLI-level tests (arg parsing, file I/O, error paths) matching this file's existing per-subcommand test convention, on top of the already-covered package-level logic (BuildShallowBootWindowRecord, DiffAddedScenarios). Fixes a dangling main.go doc-comment reference to a scripts/backfill-shallow-boot-window.sh that was never created. Does not run the live gh release upload or the actual 3-release backfill — that stays an explicit captain-flagged manual step.
Validation's detached adversarial audit found that reverting the "Download latest journey metrics artifacts" step's per-run cp target back to the flat- copy overwrite bug (the exact collision AC-2 fixes) left the full test suite green — the Go-side consumer (AggregateLedger/ReadRecordsDir) was fixture- tested, but the shell script producing the layout never was. Extends internal/release/journey_workflow_test.go's existing extractStepRun + stubbed-gh harness with a positive-path test (two stubbed runs, same record filename, different content, asserts both land in their own subdirectory intact) and an adversarial twin that reverts the cp target and asserts the two runs collapse into one file — proving the positive test actually discriminates a fixed script from a broken one.
clkao
pushed a commit
that referenced
this pull request
Jul 3, 2026
clkao
pushed a commit
that referenced
this pull request
Jul 3, 2026
…, spec-bloat trim ordered (PR #470 held)
added 3 commits
July 3, 2026 10:50
…t Minors C1 (production bug): runtime-live-e2e.yml's journey-delta-comment job hardcoded --metrics-dir to a path that doesn't match download-artifact's real nested layout (verified against run 28432388663's zip), REDing the job on every PR. Adds a "Locate this run's journey metrics" step using release.yml's own layout-agnostic find+cp pattern, tested against a realistic nested fixture. C2: BuildShallowBootWindowRecord's BaselineTokens (turns[0].Usage) does not reconstruct the removed teamRecacheSpikeFloor signal for a multi-turn pre-greet window, since the spike can land on any pre-greet turn, not just the first. Adds PreGreetPeakCacheCreation (max cache_creation across pre-greet turns) — the exact aggregate the original threshold check compared — with a regression test proving BaselineTokens alone misses a spike not on turn 0. C4: corrects BaselineTokens' documented semantics — per-turn usage is not cumulative, so only Tokens.Context() - BaselineTokens.Context() is a defensible derived quantity, not a full-struct subtraction. TRIM (captain-ordered de-scope): removes the AC-4 backfill's shipped CLI subcommands (shallow-boot-window-record, ledger-diff) and internal/release's ledgerdiff package + tests (~530 lines) in favor of a documented runbook using jq -S .scenarios | diff plus a throwaway go run script over the exported BuildShallowBootWindowRecord — the extraction/safeguard logic stays reachable, just not shipped as permanent CLI surface for a one-time procedure. Also removes three named meta-tests that verified test-fixture setup rather than product behavior. Minors: the PR delta comment now shows cache_read/cache_creation deltas separately (not just the total — they differ ~12x in cost and meaning for a boot metric); a no-baseline row renders "n/a (new)" instead of a self-delta against an implicit zero baseline; the sticky comment is now found by its marker and updated by id (gh api PATCH) instead of --edit-last, which could target the wrong comment if another automated comment interleaves.
… steps Validation's cycle-3 adversarial audit found that reverting ONLY the "Post the journey-cost delta PR comment" step's --metrics-dir back to the old broken hardcoded path left the full suite green — nothing verified the Post step stayed wired to the "Locate this run's journey metrics" step's actual output directory. Same class of gap Cycle 1 found: a step's own mechanism tested in isolation, the wiring between two steps not. Adds TestJourneyDeltaLocateAndPostStepsShareMetricsDir, which extracts both steps' real run: blocks and derives each step's directory value from the actual script text (not a hardcoded assumption), then asserts they match. Verified it catches the exact adversarial mutation validation performed. Also folds in the two minors from the same review: fixes a stale "backfill CLI" doc-comment reference in shallow_boot_window_record.go left over from the AC-4 TRIM (should read "runbook"), and adds TestRenderJourneyDeltaCommentShowsTokenClassBreakdown, which checks the PR comment's Cache Read Δ / Cache Creation Δ table cells land in the correct (non-swapped) columns — verified it catches a reversion to the old Turns/Tokens-total/Cost table shape.
…-metrics-record-and-pr-delta # Conflicts: # cmd/spacedock-release/main.go
clkao
pushed a commit
that referenced
this pull request
Jul 3, 2026
…elta, resolve PR #470 merge conflict against main
Extend JourneyDelta with DurationMSDelta, computed in ComputeJourneyDeltas from the existing journeymetrics.Record.DurationMS field, and render it as a new Duration Δ (ms) column in RenderJourneyDeltaComment.
Contributor
Journey cost delta
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces two frozen, meaningless CI thresholds with real per-PR boot-cost tracking and historical release visibility.
What changed
Evidence
e7f